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ABSTRACT 

The  hypothesis  is  advanced  that  testing  and  evaluation 
(T&E)  of  complex  weapons  systems  requires  unique  skills,  that 
the  testing  and  evaluation  of  weapons  systems  has  evolved 
into  a  recognizable  engineering  discipline,  and  that  pro- 
fessional technical  personnel  in  the  Department  of  Defense 
Test  and  Evaluation  community  should  be  considered  as  unique 
assets  and  supported  by  the  establishment  of  a  postgraduate 
curriculum  in  T&E  engineering. 

The  evolution  of  DOD  T&E  is  traced  and  analyzed  with 
particular  attention  to  capability  requirements  of  personnel. 

The  general  conclusion  is  reached  that  the  hypothesis  can 
not  now  be  universally  supported.   Reasons  for  this  position 
are  given  and  recommendations  made  for  improving  capabilities 
of  T&E  personnel. 
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I.   INTRODUCTION 

A.  GENERAL 

Test  and  Evaluation  (T&E)  is  conducted  to  support  the 
acquisition  of  weapon  systems  by  the  Department  of  Defense 
(DOD) .   These  weapon  systems  integrate  the  latest  advances 
in  technology  and  are  complex.   T&E  field  activity  personnel 
have  suggested  that  advanced  education  is  required  to  support 
the  T&E  of  modern  weapon  systems.   It  is  also  believed  that 
T&E  is  evolving  as  an  engineering  discipline/  and  a  graduate 
level  curriculum  in  T&E  engineering  should  be  established. 

B.  OBJECTIVES 

The  objectives  of  this  study  are  to: 

1.  Show  that  T&E  is  an  important  element  of  the  weapon 
systems  acquisition  cycle  and  should  be  recognized  as  a 
discipline  which  is  unique  to  DOD. 

2.  Determine  if  there  is  a  need  for  T&E  postgraduate 
education  and  if  it  should  be  institutionalized  in  the  DOD. 

3.  Propose  a  model  for  T&E  postgraduate  education  for 
military  and  civilian  personnel. 

C.  HYPOTHESIS 

The  T&E  of  complex  weapon  systems  and  sub-systems  requires 
unique  skills  to  understand,  design,  conduct,  evaluate,  and 
report  test  results.   The  T&E  of  weapon  systems  in  support  of 


the  DOD  acquisition  cycle  has  evolved  into  a  recognizable 
engineering  discipline.   The  professional  technical  personnel 
in  the  DOD  T&E  community  should  be  considered  as  unique 
assets  and  should  be  supported  by  establishing  a  postgraduate 
curriculum  in  T&E  engineering. 

D.  SCOPE 

This  research  considered  the  evolution  of  T&E  in  the  DOD 
in  the  1970 's  to  the  beginning  of  the  1980 's.   The  T&E  of  air- 
craft weapon  systems  in  the  Department  of  the  Navy  (DON)  was 
used  as  the  primary  means  for  illustrating  the  objectives  of 
this  research.   If  a  contiuum  of  need  (from  minimum  to  maximum) 
for  T&E  postgraduate  education  was  applicable,  the  T&E  of 
aircraft  weapon  systems  would  fall  in  the  maximum  need  posi- 
tion of  this  scale.   This  constraint  does  not  detract  from  the 
universality  of  the  results  to  the  role  of  T&E  in  support  of 
other  types  of  weapon  systems  acquisition  in  the  DOD.   Although 
each  of  the  Services  does  not  face  exactly  the  same  conditions 
in  acquisitions  of  weapons  systems,  the  tasks  and  problems  of 
T&E  are  similar.   Departments  of  the  Army  and  the  Air  Force 
T&E  perspectives  were  used  when  appropriate  to  this  study. 

E.  METHODOLOGY 

The  approach  used  to  prepare  this  study  was  threefold. 
The  author  relied  heavily  on  15  years  of  experience  in  the 
T&E  of  aircraft  weapon  systems.   An  archival  search  was  con- 
ducted for  references  relative  to  this  study.   Interviews 
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were  held  with  Naval  Postgraduate  School  faculty,  high  level 
DOD  officials,  and  T&E  field  activity  personnel. 

F.   REVIEW  OF  RELATED  LITERATURE 

Two  previous  Naval  Postgraduate  School  (NPS)  theses  [Refs. 
1,2]  deal  with  the  T&E  of  Navy  aircraft  weapon  systems  and 
are  related  to  this  study  topic. 

In  reference  1,  LCDR  Bowes  evaluated  the  Navy  T&E  process 
for  aircraft  weapon  systems,  identified  deficiencies  and  good 
features  of  this  process,  and  proposed  a  reorganization  of  the 
Navy's  T&E  organizational  structure.   One  of  his  conclusions 
was  the  need  for  a  course  of  instruction  in  the  discipline  of 
weapon  system  T&E.   In  addition  he  recommended  that  a  Systems 
T&E  Training  School  should  be  established  at  each  of  the  major 
test  centers  resulting  from  reorganization  of  the  Navy's  T&E 
activities.   This  thesis  presents  the  first  documented  need 
for  formal  Navy  T&E  education. 

In  reference  2,  LCDR  Stoll  examined  the  tasks  assigned  to, 
the  level  of  knowledge  and  experience  of,  and  the  need  for 
formal  education  of  project  officers  at  aviation  T&E  activities 
He  proposed  a  T&E  education  syllabus  and  alternatives  for 
achieving  it  based  on  the  results  of  a  survey  of  78  project 
officers  located  at  development  and  operational  aviation  T&E 
activities.   Seventy-three  officers  were  then  attached  to  the 
test  activities  and  five  were  attending  NPS  but  had  previous 
T&E  experience. 
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He  reviewed  test  reports  from  the  test  activities  to 
determine  the  areas  of  knowledge  and  compared  these  with  areas 
of  knowledge  obtained  from  interviews  and  questionnaires. 
Three  out  of  four  officers  had  received  inadequate  or  no  formal 
training  in  the  pertinent  subject  areas.   One  out  of  two 
project  officers,  in  their  first  year  of  experience,  indicated 
that  their  lack  of  T&E  knowledge  reduced  their  effectiveness 
to  do  their  jobs.   As  a  result  of  the  interviews  and  question- 
naires, the  pertinent  subject  areas  were  reduced  from  61  to 
49.   These  results  substantiated  the  need  for  formal  T&E 
education. 

LCDR  Stoll  designed  a  syllabus  consisting  of  10  courses 
and  110  instructor  contact  hours.   Six  alternatives  were  pro- 
posed.  The  alternatives  range  from  a  four  to  six  quarter  in- 
residence  T&E  curriculum  at  the  NPS  (least  favored  by  the 
project  officers,  since  they  would  rather  be  flying)  to  local 
programs  conducted  by  the  test  activities.   He  concluded  that 
one  approach  would  not  satisfy  the  need  for  formal  T&E  educa- 
tion, consequently,  a  combination  of  methods  was  the  recommend- 
ed approach.   An  idealized  approach  was  presented  which  con- 
sidered eventual  implementation  of  all  the  proposed  alterna- 
tives as  a  solution  to  this  dilemma. 

Since  these  theses  were  completed,  the  Naval  Air  Test 
Center  was  reorganized  and  the  Naval  Air  Systems  Command 
established  an  AIR-06  T&E  Coordinator  Division.   It's  mission 
is  to  coordinate  and  manage  the  Naval  Air  Systems  Command 
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T&E  resources  which  include  the  field  activities  and  the 
major  test  ranges.   In  addition,  the  T&E  Coordinator  integrates 
the  operational  T&E  command  test  requirements  into  the  Test 
and  Evaluation  Master  Plan  for  aircraft  weapon  systems.   The 
theses  do  not  address  the  education  requirements  of  the 
civilian  T&E  population  which  is  a  major  portion  of  the  T&E 
community. 

Reference  3  is  related  in  several  ways  to  this  study. 
Dr.  Matthews  provides  a  compilation  of  T&E  related  material 
which  he  prepared  to  focus  attention  on  the  evolving  nature 
of  T&E.   He  advocates  the  recognition  of  T&E  as  an  engineering 
discipline.   From  his  viewpoint  T&E  in  the  DOD  is  an  extension 
of  basic  scientific  and  engineering  disciplines.   One  of  his 
observations  was  that  there  is  a  need  for  formal  T&E  education 
and  he  proposed  several  alternatives  for  implementing  it.   The 
emphasis  of  this  document  is  toward  providing  a  reference 
source  for  background  information  on  T&E  in  the  DOD  and  an 
aid  to  promoting  T&E  as  an  engineering  discipline. 

G.   DEFINITIONS 

Each  of  the  Services  over  the  years  has  introduced  ter- 
minology pertaining  specifically  to  its  efforts  in  T&E.   The 
Joint  Logistics  Commanders  (consists  of  Commander  U.  S.  Army 
Materiel  Development  and  Readiness  Command,  Commander  Air 
Force  Logistics  Command,  Commander  Air  Force  Systems  Command, 
and  Chief  of  Naval  Material  Command)  was  formed  to  promote 
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cooperation  in  joint  service  acquisition  programs.   An  ad 
hoc  group  on  T&E  planning  guidance  published  the  Compendium 
of  Test  Terminology  [Ref.  4].   This  document  provides  a  use- 
ful working  level  reference  to  avoid  confusion  when  the  same 
term  is  used  by  each  Service.   The  following  definitions 
define  the  three  types  of  T&E  most  frequently  referred  to  and 
defined  in  reference  5. 

1 .  Development  Test  and  Evaluation  (DT&E) 

DT&E  is  that  T&E  conducted  to  assist  the  engineering 
design  and  development  process  and  to  verify  attainment  of 
technical  performance  specifications  and  objectives.   DT&E  is 
usually  accomplished  or  managed  by  the  DOD  Component's  material 
agency.   It  includes  T&E  components,  subsystems,  hardware/ 
software  integration,  related  software,  and  prototype  or  full- 
scale  engineering  development  models  of  the  system.   T&E  of 
compatibility  and  interoperability  with  existing  or  planned 
equipment  and  systems  are  also  included. 

2.  Operational  Test  and  Evaluation  (OT&E) 

OT&E  is  that  T&E  conducted  to  estimate  a  system's 
operational  effectiveness  and  operational  suitability,  identify 
needed  modifications,  and  provide  information  on  tactics, 
doctrine,  organization,  and  personnel  requirements.   Acquisition 
programs  shall  be  structured  so  that  OT&E  begins  as  early  as 
possible  in  the  development  cycle.   Initial  Operational  Test 
and  Evaluation  (lOT&E)  must  be  accomplished  prior  to  the 
Milestone  III  decision  (production) . 
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3.   Production  Acceptance  Test  and  Evaluation  (PAT&E) 

PAT&E  is  T&E  of  production  items  to  demonstrate  that 
procured  items  fulfill  the  requirements  and  specifications  of 
the  procuring  contract  or  agreements.   Each  DOD  Component  is 
responsible  for  accomplishing  PAT&E. 
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II.   BACKGROUND 

A .   GENERAL 

The  weapon  system  acquisition  policies  of  the  DOD  in  the 

late  1960 's  received  a  lot  of  public  attention.   DOD  had  earned 

a  reputation  for  cost  growth,  schedule  delays,  and  production 

of  systems  before  they  were  ready.   This  increased  visibility 

of  DOD's  perceived  mismanagement  of  the  defense  tax  dollar 

resulted  in  a  study  of  the  entire  organization,  structure  and 

operation  of  DOD  [Ref .  6] .   A  part  of  this  study  was  a  review 

of  the  role  of  Test  and  Evaluation  (T&E)  in  the  acquisition  of 

weapon  systems.   A  conclusion  of  the  study  was: 

"Operational  test  and  evaluation  has  been  too  infrequent, 
poorly  designed  and  executed,  and  generally  inadequate." 
[Ref.  6:2] 


To  focus  additional  attention  on  test  and  evaluation  the  study 
recommended  a  Defense  Test  Agency  which  would  oversee  all 
Defense  T&E.   Specifically  it  would  design  or  review  the 
designs  for  tests;  monitor  the  entire  Defense  test  program; 
and  conduct  T&E  of  systems  which  can  be  used  across  Services. 
Also,  the  Panel  recommended  that  a  separate  funding  category 
should  be  identified  for  T&E  and  operational  testing.   The 
Defense  Test  Agency  would  have  responsibility  for  this 
funding. 

Deputy  Secretary  of  Defense,  David  Packard,  was  instru- 
mental in  implementing  recommendations  of  the  Blue  Ribbon 
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Defense  Panel  which  have  affected  T&E.   He  established  the 
Director  of  Defense  T&E  and  moved  the  Services  to  establish 
independent  test  agencies  which  report  to  the  Service  Chiefs 
[Ref .  7] .   Additionally,  he  increased  the  role  of  OT&E  results 
in  the  acquisition  process  by  issuing  DOD  Directive  5000.1 
[Ref.  8]  requiring  that  OT&E  be  conducted  before  a  decision 
is  made  on  full  scale  production.   Congress  also  gave  momentum 
to  the  changing  role  of  OT&E  by  passing  a  law  which  requires 
that  DOD  give  Congress,  annually,  the  results  of  OT&E  with 
its  request  for  funds  on  systems  it  is  planning  to  procure 
[Ref.  9].   The  DOD  Directive  5000.3,  Test  and  Evaluation 
[Ref.  7] ,  was  issued  to  define  policy  on  the  types  of  testing 
and  the  roles  of  the  Services  in  T&E.   This  directive  also 
established  the  requirement  for  a  Test  and  Evaluation  Master 
Plan  (TEMP)  which  coordinates  all  the  testing  requirements 
for  a  given  program. 

The  Defense  Science  Board  Task  Force  on  T&E  was  formed  in 
late  1972.   It  was  tasked  to  study  a  group  of  major  weapon 
systems  to  determine  whether  improvement  in  T&E  would  permit 
earlier  discovery  of  problems  and  initiation  of  corrective 
actions.   Its  report  [Ref.  10]  in  early  1974  identified  that 
improvements  were  needed  in  test  planning  and  scheduling.   T&E 
participation  was  frequently  deleted  due  to  overruns  of  time 
and  money  in  other  phases  of  the  program.   The  Panel  described 
a  T&E  gap  which  occurred  when  testing  pre-production  hardware 
or  prototypes.   Since  the  decision  to  start  production  cannot 
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occur  until  after  initial  OT&E  (lOT&E)  is  completed,  a 
substantial  lag  (1.5  to  2  years)  exists  between  completion 
of  development  testing  and  manufacture  of  the  first  produc- 
tion article.   The  Panel  also  provided  nine  related  reports 
which  provide  T&E  guidelines  for  different  types  of  systems, 
for  example,  for  missile  weapon  systems  and  for  aircraft 
systems. 

The  Office  of  Management  and  Budget  published  0MB  Circular 
A-109  in  1976  which  outlined  the  government's  policy  on  the 
acquisition  of  major  systems  [Ref.  11].   This  circular  was 
issued  to  curb  cost  overruns  and  ensure  the  new  systems  are 
needed.   As  a  result  of  the  issuance  of  this  circular  the  DOD 
and  Service  directives  and  instructions  had  to  be  revised.   No 
major  changes  in  T&E  policy  were  generated  except  for  the 
reemphasis  of  the  requirement  for  T&E  criteria.   This  included 
the  concept  of  independent  testing  and  testing  in  the  opera- 
tional environment  prior  to  approval  for  full  production. 

The  lengthening  of  the  acquisition  cycle  was  studied  by 
a  Task  Force  of  the  Defense  Science  Board  and  reported  in  197  8 
[Ref.  12] .   It  concluded  that  sequential  and  separate  testing 
of  the  systems  by  the  test  activities  of  the  developer  and  the 
independent  test  activities  representing  the  user  often  adds 
unnecessary  costs  and  time.   Thus  the  Tas]<:  Force  recommended 
conducting  combined  development  and  operational  tests  wherever 
feasible  and  separate  evaluations  of  the  data.   The  Task  Force 
expressed  concern  that  independent  test  activities  with  their 


increased  influence  would  demand  duplicate  facilities  and 
increased  testing.   What  is  really  desireable  is  combined 
within  the  same  service)  testing  and  independent  evaluation. 
AlsO/  it  was  felt  that  the  T&E  activities  should  participate 
throughout  the  entire  acquisition  cycle. 

B .   ENVIRONMENT 

Presently,  two  major  Navy  aircraft  weapon  systems  are  in 
full  scale  development  testing  at  the  Naval  Air  Test  Center, 
Patuxent  River,  Maryland.   The  F-18  program  is  employing  the 
single-site  test  concept.   The  contractor  performs  its  develop- 
ment and  operational  testing  at  this  government  facility  with 
Navy  test  activities  monitoring  the  contractor's  tests.   At 
specified  times  each  of  the  Navy's  test  activities  conduct 
testing  and  perform  independent  evaluations  of  the  data. 
This  approach  has  numerous  advantages  and  disadvantages  which 
have  not  been  fully  assessed. 

The  second  major  aircraft  weapon  system  in  full  scale 
development  testing  at  the  Naval  Air  Test  Center  is  the  Light 
Airborne  Multi-Purpose  System  (LAMPS)  MK-III.   The  LAMPS 
MK-III  helicopter  and  airborne  systems  are  an  extension  of  the 
ship's  Anti-Submarine  Warfare  (ASW)  capability.   In  the  case 
of  the  LAMPS  MK-III  the  Naval  Air  Test  Center  is  not  a  single 
site  test  location.     It  functions  as  the  focal  point  for  all 
the  Navy  field  activities'  participation  in  the  testing  of 
the  complete  LAMPS  MK-III  weapon  system  (includes  aircraft  and 
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ship  systems) .   The  responsible  operational  test  activity  is 
participating  in  combined  testing  with  the  developer's  test 
activities  to  reduce  duplication,  time,  and  costs  of  testing. 
The  operational  test  activity  still  retains  designated  test 
periods  for  Initial  OT&E  (lOT&E)  and  Follow-on  OT&E  (FOT&E) . 
The  Test  and  Evaluation  Master  Plan  (TEMP)  was  utilized  as  a 
major  management  tool  in  working  out  the  participation  of  the 
T&E  activities  in  this  program.   The  flow   f  instructions 
which  govern  the  preparation  of  the  TEMP  is  shown  in  Appendix 
A,  Figure  1. 

The  TScE  participation  in  the  minor  acquisition  programs 
normally  does  not  fare  very  well.   It  is  estimated  that 
minor  acquisition  programs  amount  to  three  times  the  major 
programs.   The  management,  however,  of  these  programs  is  not 
normally  provided  with  sufficient  resources  or  experience  to 
manage  the  acquisition  until  it  runs  into  trouble.   The 
Services  have  initiated  the  same  acquisition  procedures  for 
minor  programs  as  for  major  programs  with  lower  level  review 
and  approval  requirements.   Consequently,  the  preparation  of 
a  TEMP  to  coordinate  the  TSeE  effort  is  an  exception  rather 
than  a  rule.   Few,  if  any  program  management  offices,  have 
personnel  experienced  in  T&E.   The  TEMP  is  a  useful  test 
coordinating  document,  because,  it  brings  the  players  together 
to  negotiate  their  requirements.   It  must  be  remembered, 
however,  that  the  acquisition  cycle  is  dynamic,  and  the  TEMP 
is  a  point-in- time  document.   The  program  office  must  therefore 
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become  innovative  in  its  effort  to  communicate  changes  in 
the  TEJyiP  to  all  the  participating  activities. 

C .   TECHNOLOGY 

The  major  and  minor  acquisition  programs  normally  incor- 
porate the  latest  technological  advances.  American  industry 
is  the  world  leader  in  developing  new  technology  and  in 
bringing  it  to  the  market  place.  High  technology  is  driving 
the  economy  while  all  segments  of  industry  scramble  to  apply 
it  to  increase  productivity  and  drive  down  production  costs. 
This  same  pyschology  exists  in  the  acquisition  of  weapon 
systems. 

The  T&E  community  personnel  are  dealing  with  increasingly 
sophisticated  and  complex  equipment  and  systems.   The  use  of 
airborne  computers  and  associated  software  have  found  applica- 
tion in  on-board  processing  of  information  and  analysis  of 
acoustical  data  in  ASW  missions,  navigation,  weapons  delivery 
control,  communication,  etc. .   Rapid  advances  in  semi-conductor 
devices  continues  at  a  rapid  rate.   Functions  which  previously 
could  not  be  accomplished  because  of  the  size  of  the  resulting 
hardware  are  now  being  incorporated  into  airborne  platforms. 
Flight-control  systems  are  now  built  around  airborne  computers 
vice  mechanical  linkages.   How  the  aircraft  flies  is  a  function 
of  the  application  of  complex  mathmatical  control  theory  and 
programming  skills. 
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Airborne  Command,  Control,  Communication  and  Intelligence 

3 

(C  I)  functions  are  performed  airborne  as  well  as  on  the 

ground.   Achievement  in  this  area  has  leaped  forward  with  the 
incorporation  of  semi-conductor  advances  into  airborne  radar 
and  communications  devices.   Likewise  the  electronic  warfare 
capabilities  to  detect,  jam, or  fool  the  agressor  have  all  been 
enhanced  by  the  incorporation  of  semi-conductor  technology. 

Material  technological  advances  in  ceramics,  composite 
and  metal  compounds  are  being  used  in  airframes  to  increase 
strength  to-weight-ratios,  reduce  detectability,  and  to 
increase  reliability  and  maintainability.   Technological 
advances  in  ceramics  and  metal  compounds,  producing  higher 
temperature  resistant  materials,  are  being  applied  to  propul- 
sive devices.   As  a  result,  jet  engines  are  producing  higher 
thrust-to-weight  ratios  and  are  more  energy  efficient. 

The  technology  list  that  the  T&E  community  must  deal  with 
is  lengthy.   The  T&E  of  these  high- technology  systems  presents 
unique  testing  challenges.   The  challenge  facing  the  T&E 
community  is  to  stay  technically  competent  to  do  effective 
testing  in  the  face  of  fixed  personnel  ceilings  and  increasing 
workload. 

D.   ORGANIZATION  STRUCTURE 

The  Secretary  of  Defense  (SOD)  established  the  Director 
of  Defense  for  Test  and  Evaluation  (DDT&E) .   The  DDT&E  was 
designated  to  report  through  the  Under  Secretary  of  Defense 
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for  Research  and  Engineering.   The  DDT&E  was  identified  as 
having  responsibility  for  monitoring  and  advising  the  SOD  on 
T&E  conducted  by  the  Services.   It  also  provides  access  to 
the  office  of  the  SOD  for  the  Services'  OT&E  activities  on 
matters  concerning  independent  testing.   It  is  unlikely  that 
the  military  OT&E  activities  would  use  this  access  without 
going  through  their  normal  command  structures  first.   The 
DDT&E  organizationally  is  also  the  focal  point  for  joint 
service  testing  programs. 

The  organizational  changes  which  resulted  from  the  increased 
emphasis  on  OT&E  had  the  least  impact  on  the  Navy.   The  Navy 
already  had  the  Operational  Test  and  Evaluation  Force  (OPTEVFOR) 
in  being.   Therefore,  it  was  able  to  comply  rapidly  to  Deputy 
Secretary  of  Defense  Packard's  request  for  the  Services  to  set- 
up independent  operational  testing  agencies.   The  Navy  establish- 
ed a  Director  of   RDT&E,  OP-98,  with  the  Commander  of  OPTEVFOR, 
OP-98  3,  reporting  to  the  Navy  Director  of   RDT&E.   This 
organizational  change  was  the  Navy's  compliance  for  the 
operational  test  activity  to  report  to  the  Service  Chief 
independently  of  the  developing  agency. 

The  Naval  Air  Systems  Command  organizational  structure  was 
amended  in  1975.   An  Assistant  Commander  for  Test  and  Evaluation 
Coordination  was  established  [Ref.  13].   This  organization  had 
existed  previously  at  a  lower  organizational  level  and  was 
tasked  with  allocating  the  Navy's  RDT&E  aircraft  resources. 
This  organizational  change  was  partially  attributed  to  the 
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findings  of  the  Donaldson  Committee  Report  on  the  T&E  base 
study,  conducted  for  the  Commander  Naval  Air  Systems  Command 
[Ref.  14],   A  similar  type  of  organization  for  TSeE  coordina- 
tion is  not  used  by  the  other  Services. 

The  Army  and  the  Air  Force  required  several  years,  until 
the  mid  1970 's  to  comply.   The  Army  established  the  Operational 
Test  and  Evaluation  Agency  (OTEA)  and  the  Air  Force  set  up  the 
Air  Force  Test  and  Evaluation  Center  (AFTEC) .   The  Commander 
of  OTEA  reports  to  the  Army  Chief  of  Staff  and  the  Commander 
of  AFTEC  reports  to  the  Chief  of  Staff  Air  Force.   Both  OTEA 
and  AFTEC  have  small  headquarters'  staffs  which  are  dependent 
on  the  operating  commands  to  provide  the  manpower  resources 
to  support  OT&E.   The  Army  and  Air  Force  started  out  emphasiz- 
ing combined  testing  in  order  to  reduce  costs  and  to  use  pre- 
production  resources  more  efficiently. 

E .   PLAYERS 

1 .   Program  Manager 

The  program  manager  is  the  hub  of  the  acquisition 

cycle  and  the  key  to  the  level  of  T&E  participation.   He  must 

budget  and  plan  for  this  participation  and  provide  for  any 

special  resources  such  as  facilities  which  could  involve 

military  construction  funding.   Normally  the  program  manager 

changes  many  times  during  major  weapon  system  acquisitions. 

Since  it  takes  from  12  to  15  years  from  concept  initiation  to 

initial  operational  capability,  it  is  not  unusual  for  a  program 

to  have  five  program  managers. 
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The  program  managers  seldom  have  previous  T&E  experience 
and  it  is  easy  for  an  adversary  relationship  to  develop  between 
the  program  manager  and  the  test  community.   He  frequently  does 
not  understand  the  functions  of  the  Major  Ranges  or  the  Test 
Centers  and  how  to  use  them  to  support  the  development  effort 
or  reduce  engineering  risk.   The  test  activities  by  reporting 
what  is  wrong  with  the  system  increase  the  program  office 
workload  to  explain  away  the  problems  or  to  plan  for  funding 
and  schedule  changes  to  correct  the  deficiencies. 

The  military  specifications  and  policy  instructions 
which  are  part  of  most  weapon  system  contracts  have  built-in 
test  requirements  which  often  come  as  a  surprise  to  the 
inexperienced  program  manager.   Viewing  T&E  from  the  program 
manager's  perspective,  his  main  objective  is  to  get  the  OT&E 
activity  to  grant  a  Provisional  Acceptance  for  Service  Use 
(PASU)  or  an  Acceptance  for  Service  (ASU)  which  is  the  more 
desireable.   In  the  real  world  of  politics,  funding  availability, 
and  schedule  crunches  the  tendency  is  to  make  decisions  favor- 
ing OT&E  requirements  at  the  expense  of  DT&E  requirements. 

Program  managers  for  minor  programs  are  normally  even 
less  prepared  than  the  managers  of  major  programs.   Even 
though  the  minor  programs  account  for  approximately  three  times 
the  funding  of  major  acquisition  programs,  the  program  manager 
is  frequently  a  technical  type  who  has  not  developed  the 
management  and  technical  competencies  to  conduct  a  minor 
acquisition.   He  must  write  the  technical  requirement  portion 
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of  the  Request  for  Proposal,  evaluate  proposals,  write  the 
work  statement,  provide  technical  direction  to  the  development 
effort,  write  test  requirements,  plan  the  conduct  of  the 
engineering  development  tests,  and  provide  guidance  to  higher 
management.   The  minor  program  manager  historically  is  compet- 
ing with  the  major  programs  for  funds  inadequate  to  support 
the  procurement  of  all  the  items  in  deveopment.   Testing 
appears  as  an  expensive  item  in  a  limited  funding  situation 
or  in  an  inadequately  planned  program  and  is  usually  among  the 
first  items  to  be  compromised.   Minor  programs,  due  to  lower 
visibility,  are  even  more  susceptible  to  frequent  program 
manager  turnover.   One  of  the  findings  of  a  GAO  report  con- 
ducted on  the  Defense  System  Acquisition  Review  Council  (DSARC) 
was  a  relatively  rapid  turnover  of  high  ranking  DOD  and  Service 
personnel  responsible  for  the  integrity  of  the  DSARC  process 
[Ref .  15] .   High  ranking  DOD  civilians  and  military  personnel 
were  changed  at  an  average  of  28  months  and  24  months  respec- 
tively.  This  data  certainly  reflects  unfavorably  on  the 
management  of  the  acquisition  cycle  and  is  relatively  representa- 
tive of  program  manager  turnover.   Unless  the  T&E  program  is 
documented  in  a  TEMP  in  the  early  phases  of  the  acquisition 
cycle,  T&E  is  always  vulnerable  to  inadequate  funding  and 
support. 

2.   DT&E  Testers 

The  contractor  for  the  weapon  system  conducts  testing 
during  the  acquisition  to  support  engineering  development 
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efforts.   The  government  personnel  employed  by  the  Major 
Ranges  and  Test  Facility  Base  (MRTFB)  are  primarily  respon- 
sible for  conducting  the  government's  technical  testing  during 
the  full  scale  development  and  production  phases  of  the 
acquisition  cycle  (Appendix  A,  Figure  2).   The  developer's 
test  activities  also  participate  in  the  earlier  phases  of  the 
acquisition  cycle  but  at  a  lesser  degree  than  laboratory  or 
development  center  personnel.   A  listing  of  MRTFB  and  other 
activities  involved  in  T&E  are  listed  in  Appendix  A. 

It  is  difficult  to  determine  the  exact  number  of 
government  personnel  employed  in  T&E.   The  civilian  ceilings 
have  declined  some  in  recent  years,  however,  with  an  increas- 
ing work  load  the  test  activities  have  turned  to  contracting 
to  operate  test  ranges  and  for  some  engineering  support. 
Collectively  the  DOD  RDT&E  community  totals  in  excess  of 
100,000  personnel  [Ref.  16].   The  DDT&E  estimates  that  approx- 
imately 50,000  military,  civilian,  and  contracted  for  personnel 
operate  the  MRTFB  and  provide  special  test  support.   These 
figures  do  not  include  OT&E  personnel  such  as  those  assigned 
to  COMOPTEVFOR  and  the  operating  forces  assigned  temporarily 
to  the  Army  and  Air  Force  operational  test  activities. 

The  MRTFB  is  staffed  with  both  military  and  civilian 
engineers  and  scientists.   It  is  estimated  that  they  will 
spend  $1.1  billion  in  FY  1981  [Ref.  17]  not  including  the  use 
of  Operating  and  Maintenance  (O&M)  funds  —  O&M  funds  support 
the  weapon  system  once  it  is  in  production  to  the  end  of  its 
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life  cycle  —  to  conduct  T&E  of  weapon  systems.   In  terms  of 
government  personnel  and  funds  T&E  is  big  business. 

The  DT&E  community  in  the  past  has  had  difficulty  in 
obtaining  engineering  and  scientific  personnel  due  to  a 
combination  of  factors.   A  declining  number  of  graduates  in 
engineering  and  the  sciences,  remote  locations  of  the  DT&E 
activities,  reduced  personnel  ceilings,  hiring  freezes  which 
frustrate  recruiting  efforts,  and  starting  salaries  substant- 
ially below  industry  entry  levels  have  summed  to  weaken  the 
DT&E  technical  base.   Military  personnel,  specifically  Navy, 
with  previous  T&E  experience  are  less  likely  to  return  for  a 
second  tour  of  the  DT&E  activities  due  to  a  lack  of  a  defined 
career  path  in  T&E.   This  is  especially  true  in  the  DT&E  of 
Navy  aircraft  weapon  systems.  Costly  and  extensive  training 
is  provided  to  naval  flight  officers  to  develop  engineering 
test  pilot  skills.   In  turn  these  officers  serve  an  approximately 
two-year  tour  before  moving  on  to  an  assignment  unrelated  to 
their  T&E  skills. 

DT&E  skills  are  mostly  learned  by  on-the-job  training 
(OJT) .   Each  of  the  DT&E  activities  have  established  the 
methodologies  and  techniques,  instrumentation,  and  facilities 
to  acquire  hard  data  to  make  assessments  of  engineering  develop- 
ment and  risk.   Most  of  the  DT&E  activities  are  in  remote 
areas  and  are  limited  markets  for  advanced  technical  education. 
This  has  constrained  the  access  of  the  DT&E  personnel  to 
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advanced  technical  education  needed  to  renew  their  engineering 
skills  to  challenge  the  complexity  of  modern  weapon  systems. 

The  Navy  DT&E  of  aircraft  weapon  systems  uses  skilled 
military  operator  personnel.   Even  though  DT&E  is  conducted 
under  more  controlled  conditions  than  operational  testing, 
military  test  personnel  are  qualified  to  relate  test  results 
to  expected  performance  under  fleet  operating  conditions. 
The  DTScE  results  are  reported  to  the  development  agency  and 
OT&E  personnel  are  encouraged  to  use  DT&E  results  where  feasible, 
Also,  DT&E  personnel  are  required  to  support  OT&E.   In  contrast 
OT&E  results  are  reported  to  the  Chief  of  Naval  Operations 
level  and  are  not  normally  available  to  DT&E  personnel.   The 
reporting  of  weapon  system  deficiencies  by  DT&E  personnel  are 
not  necessarily  reported  in  the  same  form  as  by  OT&E  personnel. 
3.   OT&E  Testers 

T&E  is  a  team  effort  consisting  of  the  program  manager, 
the  development  test  activities  and  the  operational  test 
agencies.   Most  of  the  focus,  however,  in  the  1970 's  has  been 
on  an  increase  of  the  role  of  operational  testing.   The 
acquisition  policy  has  been  changed  to  include  participation 
of  OT&E  personnel  throughout  the  acquisition  cycle.   The 
Services  have  used  different  approaches  to  meet  the  increased 
responsibilities  of  OT&E.   The  Navy  has  its  operational  test 
agency  headquarters  in  Norfolk,  Virginia.   It  has  staffs  on 
the  East  and  West  Coast,  three  operational  aircraft  squadrons 
(based  on  type  of  aircraft  and  mission) ,  and  detachments  — 
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as  required  to  support  new  tactics,  operating  procedures,  and 
techniques.   The  Army's  operational  test  agency  is  located  at 
Falls  Church,  Virginia  and  the  Air  Force  operational  test 
center  is  located  at  Kirtland  Air  Force  Base,  New  Mexico.   The 
Air  Force  and  Army  staffs  are  small  in  comparison  with  the 
Navy  (less  than  100  personnel  vice  more  than  14  00  for  the 
Navy).   None  of  the  Service  OT&E  agencies  employ  many  civilians, 
therefore,  continuity  in  the  lengthly  acquisition  cycle  is 
difficult  to  maintain  unless  military  reassignments  are  made 
less  frequently.   The  Navy  does  not  provide  for  extended  tours 
for  military  personnel  assigned  to  OPTEVFOR.   In  fact  this 
would  be  contrary  to  the  policy  stated  by  COMOPTEVFOR  [Ref .  18] . 
This  memo  expresses  the  importance  of  having  skilled  operators 
with  recent  "blue  water"  experience  vice  highly  trained 
technical  operators  to  conduct  operational  testing.   The  Navy 
OT&E  concept  was  used  by  the  Blue  Ribbon  Panel  as  an  example 
for  the  other  Services  to  consider.   The  Navy  OT&E  was  faulted 
for  its  inability  to  produce  "hard  numbers"  to  assist  high 
level  decision  making.   The  COMOPTEVFOR  has  employed  a  few 
civilian  analysts  to  improve  on  this  situation. 

The  GAO  in  197  7  began  a  review  of  the  Service  OT&E 
efforts.   The  Navy,  having  the  most  experience  with  an  OT&E 
organization,  was  reviewed  first  [Ref.  19] .   GAO  found  the 
Navy  was  still  making  production  decisions  with  incomplete 
OT&E  results.   OT&E  could  not  be  conducted  on  some  systems  due 
to  a  lack  of  test  resources,  and  OT&E  conducted  on  systems, 
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after  production  approval  was  granted,  found  performance 

problems  that  should  have  been  corrected  before  production 

approval. 

OT&E  agencies  feel  they  are  the  only  test  activities 

which  test  the  complete  weapon  system.   OPTEVFOR  views  are 

expressed  on  this  position  in  the  Operational  Test  Director 

Guide  [Ref .  20] .   This  manual  also  raises  an  interesting  point 

on  the  OPTEVFOR  philosophy  on  the  availability  of  operational 

test  data: 

"In  some  programs  it  may  be  convenient  (or  absolutely 
necessary)  to  use  DA  field  agencies  to  get  OT&E  data 
reduced  and,  to  some  degree,  analyzed.   In  these  situa- 
tions it  is  mandatory  that  these  people  be  under  the 
operational  control  of  COMOPTEVFOR  (represented  by  the 
OTC/OTD/Program  Analyst)  while  they  are  working  on  OT&E 
data.   Their  work  is  defined  in  advance,  and  their  results 
are  furnished  only  to  COMOPTEVFOR,  unless  COMOPTEVFOR 
has  specifically  approved  a  wider  distribution."  [Ref.  20: 
16-2) 

Field  agencies  are  required  to  provide  all  information  to 

OT&E  personnel  and  it  could  be  implied  from  the  above  that  the 

exchange  of  data  is  in  one  direction  only. 

Although  COMOPTEVFOR' s  earlier  position  was  to  shun 

the  use  of  operator  personnel  with  technical  backgrounds,  the 

extreme  pressures  to  make  production  decisions  using  "hard 

data"  has  weakened  this  position.   In  the  OT&E  of  aircraft 

weapon  systems  there  is  trend  toward  OPTEVFOR  project  personnel 

having  technical  backgrounds.   This  has  the  potential  to 

change  the  OT&E  viewpoint  toward  acquiring  more  hard  data  to 

substantiate  OT&E  results;  and  it  raises  issues  on  duplication 

between  OT&E  and  the  developing  agency's  test  activities. 
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The  Air  Force  and  the  Army  have  approached  OT&E 
differently  than  the  Navy.   They  have  emphasized  combined 
test  activities  of  the  developmental  and  operational  testers, 
the  sharing  of  limited  pre-production  resources,  and  the  use 
of  highly  skilled  operational  testers.   The  GAO  has  critisized 
their  policies  on  OT&E  [Refs.  21,22],   In  particular,  both 
Services  needed  to  conduct  operational  testing  in  environments 
more  representative  of  the  operational  environment.   Also,  the 
Services  should  designate  distinct  periods  for  operational 
testing  and  provide  the  necessary  test  resources.   The  test 
personnel  and  support  should  be  more  representative  of  the 
type  used  when  the  system  is  deployed. 

4.   Board  of  Inspection  and  Survey  and  AIR-06 

Two  agencies  which  are  unique  to  the  Navy  are  the 
Board  of  Inspection  and  Survey  (BIS)  and  the  Naval  Air  Systems 
Command  (AIR-06) ,  the  Assistant  Commander  for  Test  and  Evalua- 
tion.  The  President  of  BIS  is  located  in  Washington,  D.  C. 
and  has  Sub- Boards  of  Inspection  and  Survey  located  in  Virginia, 
California,  and  Maryland.   The  Sub-BIS  for  aircraft  is  located 
at  Patuxent  River,  Maryland.   It  tasks,  for  example,  the  Naval 
Air  Test  Center  to  conduct  technical  tests  of  production  air- 
craft weapon  systems  or  follow  on  tests  on  significant  changes 
to  aircraft  in  use.   Based  on  the  test  results  the  President 
of  BIS  recommends  to  the  Secretary  of  the  Navy  whether  the 
aircraft  weapon  system  should  be  accepted  for  fleet  use. 
The  potential  conflict  or  duplication  of  the  BIS  and 
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COMOPTEVFOR  roles  have  been  actively  discussed  and  analyzed 
recently.   The  Navy  expressed  confirmation  of  the  need  for 
each  organizations'  function,  role  and  contribution  to  the 
T&E  process  through  the  recent  revision  of  OPNAVINST  3960. lOA 
[Ref.  23]. 

The  AIR-0  6  organization  evolved  from  the  need  to 
provide  a  focal  point  for  NAVAIRSYSCOM  RDT&E  resources.   AIR-06 
coordinates  the  Navy's  Major  Range  and  Test  Facility  Base 
(MRFTB)  with  DDT&E.   Since  the  TEMP  has  become  a  key  document 
for  major  weapon  systems,  AIR-06  has  taken  responsibility  for 
coordinating  the  DT&E  and  OT&E  requirements  for  the  program 
manager.   AIR-06  also  coordinates  the  workload  going  to  the 
NAVAIR  field  activities  with  the  resources  required  to  conduct 
DT&E.   The  effectiveness  of  this  organization  in  performing 
its  mission  has  not  been  confirmed  since  its  establishment  in 
1975. 

F.   EVOLUTION 

There  is  presently  a  significant  variance  between  the  Navy 
approach  to  T&E  and  those  of  the  Air  Force  and  Army.   In  the 
1970' s  the  decision  making  processes  for  major  acquisition  of 
weapon  systems  has  varied  from  decentralization  to  the  Services 
and  then  a  return  to  centralization  at  the  Office  of  the 
Secretary  of  Defense.   Presently,  the  trend  is  toward  account- 
ability and  decentralization  to  the  Services  as  outlined  in  a 
recent  Deputy  Secretary  of  Defense  memo  [Ref.  24] .   Consequently, 
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high  visibility  issues,  such  as,  productivity  improvement  by 
increasing  DOD  effectiveness  and  efficiency  through  reduction 
in  duplication  of  Service  efforts  will  require  more  cooperation 
between  the  Services  and  more  joint  Service  acquisition  programs. 
This  will  magnify  the  differences  in  the  Services'  approaches 
to  T&E  and  result  in  a  buildup  of  forces  toward  more  continuity 
in  test  techniques,  methodologies,  test  personnel  qualifications, 
and  analytical  techniques.   The  pressure  by  DOD  for  hard 
operational  test  data  will  increase.   This  will  force  the  Navy 
to  make  substantial  compromises  in  its  past  policies  on  indep- 
endent operational  testing  and  its  need  for  more  analysis  in 
the  evaluation  phase.   The  Services  will  approach  a  common 
TScE  base  to  allow  DOD  to  make  rapid  management  decisions  to 
reduce  the  length  of  the  acquisition  cycle.   DDT&E's  future 
role  is  unclear  in  the  present  move  toward  decentralization. 
Since  at  present  DDT&E  operates  primarily  in  a  persuasive 
mode  with  the  Services,  increased  joint  Service  programs 
might  necessitate  more  authority  for  the  DDT&E.   A  Defense 
Test  Agency  could  even  result  [Refs.  6,16]. 
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III.   GRADUATE  EDUCATION 

A.   GENERAL 

The  need  for  full-time,  fully-funded  graduate  education 
for  DOD  personnel  has  received  considerable  attention  in  the 
1970 's.   The  declining  enrollments  in  higher  education  has 
accelerated  the  availability  of  graduate  education  at  DOD 
facilities  as  colleges  and  universities  have  competed  for 
this  source  of  revenues.   Advances  in  technology  and 
society's  acceptance  of  the  value  of  education  has  increased 
this  need  for  continuing  education  by  the  work  force.   Most 
who  seek  advanced  education  do  so  to  receive  a  graduate  degree, 
Military  positions  have  been  established  requiring  advanced 
degrees  to  perform  the  duties  of  the  job.   This  frequently 
has  been  done  to  enhance  retention,  increase  the  morale  of 
the  personnel  and  provide  for  increased  job  opportunities 
after  retirement.   On  the  civilian  side,  the  bureacracy  has 
used  advanced  degrees  as  a  screening  mechanism  in  selection 
for  promotion  and  to  increase  the  technical  status  of  the 
organization. 

Before  most  personnel  make  a  commitment  to  long  term 
training  one  of  their  chief  concerns  is  whether  the  award 
of  a  graduate  degree  can  be  achieved.   Since  advanced  degrees 
are  becoming  a  norm  in  our  society,  it  is  unlikely  there  will 
be  many  participants  in  a  full-time  advanced  education  program 
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just  for  the  benefit  of  the  additional  training.   The  selec- 
tion of  civilians  for  long-term  fully-funded  training,  as  a 
recognition  for  having  high-potential  to  be  future  hign-level 
leaders  within  DOD ,  places  additional  pressure  on  the  recipient 
to  earn  an  advanced  degree.   The  sponsoring  activities  also 
subtly  encourage  that  the  advanced  degree  is  an  expected 
result  of  the  trainee's  efforts.   The  T&E  activities,  priir.arily 
those  involved  in  development  testing  and  evaluation,  have 
long  recognized  and  voiced  the  need  for  graduate  education  in 
support  of  the  T&E  process.   The  engineering  schools  have 
been  reluctant  to  offer  off-campus  programs  which  lead  to 
advanced  degrees  mainly  for  traditional  and  quality  of  gradu- 
ate education  reasons.   Most  T&E  activities  are  in  remote 
locations  and  provide  a  small  market.   This  does  not  motivate 
the  engineering  schools  to  be  flexible  in  their  attitudes 
toward  serving  the  needs  of  the  T&E  community.    An  MIT  study 
found  there  was  little  research  available  to  evaluate  the 
performance  of  engineering  schools  in  meeting  the  education 
needs  of  their  graduates  [Ref .  25] .   Most  engineering  schools 
have  been  too  preoccuppied  with  spending  Federal  research 
dollars  or  graduating  students  to  meet  industry's  requirements. 
They   have  consequently  failed  to  solicit  graduate  feedback. 
The   Accrediting  Board  for  Engineering  and  Technology,  in 
order  to  prevent  stagnation  in  engineering  education,  does 
not  advocate  rigid  standards  for  accreditation  and  encourages 
well-planned  curriculum  experimentation  [Ref.  26] . 
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The  literature  reviews  cited  in  the  Introduction  proposed 
programs  for  graduate  education  in  T&E.   The  Pacific  Missile 
Test  Center,  a  Navy  T&E  field  activity,  located  at  Point 
Mugu,  California  has  also  proposed  some  innovative  programs 
to  provide  graduate  education  to  its  technical  employees. 
The  Naval  Postgraduate  School  (NPS)  in  Monterey,  California 
has  two  courses  listed  in  its  catalog  which  are  titled  Test 
and  Evaluation  which  are  taught  by  the  Operations  Research 
Department.   In  the  past,  informal  discussions  with  Navy 
field  activities'  personnel  have  resulted  in  NPS  spokesmen 
proposing  "strawmen"  T&E  curricula.    These  proposals  have 
not  moved  forward  due  to  lack  of  Department  of  the  Navy 
support  and  current  constraints  on  NPS  resources.   The  Naval 
Aviation  Executive  Institute  Program  was  established  at  Naval 
Missile  Test  Center,  Point  Mugu,  California  in  1975  and 
moved  to  the  NPS  in  1976.   This  program  has  high  potential 
to  respond  to  the  graduate  education  needs  of  the  Navy's 
T&E  community  and  could  provide  a  model  for  the  rest  of  the 
T&E  community. 

Some  observations  are  presented  on  the  status  of  T&E 
engineering  as  a  profession.   The  parameters  which  have 
been  identified  that  pertain  to  T&E  of  aircraft  weapon 
systems  but  are  relevant  to  the  T&E  of  other  weapon  systems 
and  equipment  are  discussed.   The  needs  of  different  levels 
of  personnel  in  the  Navy  T&E  community  are  reviewed  and 
a  framework  presented  which  could  support  T&E  educational 
needs . 
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B.   FULL-TIME,  FULLY-FUNDED 

The  atmosphere  for  full-time,  fully-funded  graduate 
education  in  the  DOD,  has  not  been  very  favorable.    The  NPS 
which  in  the  eary  1970 "s  had  an  enrollment  of  1200  Navy 
students  has  had  its  budget  cut  by  Congress  to  where  the  NPS 
had  400  students  in  FY  1977.   This  figure  has  increased  to 
600  Navy  students  for  FY  81  and  the  Congressional  committees 
are  in  support  of  this  level.   The  Navy  and  Air  Force  have 
been  critized  for  the  cost  per  student  to  provide  postgraduate 
education  for  their  personnel  compared  with  the  cost  per 
student  of  using  non-military  institutions.   The  Navy  has 
been  the  smallest  user  of  public  sector  colleges  and  universi- 
ties to  provide  full-time,  fully- funded  education.   The  Air 
Force,  although  it  supports  the  Air  Force  Institute  of 
Technology  (AFIT) ,  sends  a  majority  of  its  full  time  graduate 
students  to  the  public  sector  institutions.    The  Army  sends 
most  of  its  full  time  students  to  the  public  sector  institu- 
tions except  for  those  it  sends  to  NPS  and  AFIT. 

The  GAO  [Ref.  27]  addressed  the  permissive  attitude  of 
the  services  toward  full-time,  fully-funded  graduate  educa- 
tion for  officers.   The  services  were  tasked  to  validate 
positions  requiring  advanced  education  in  order  to  perform 
the  duties  of  the  position.   These  were  primarily  positions 
in  the  science  and  technology  areas.   The  intent  of  providing 
the  postgraduate  education  was  to  prepare  officers  to  function 
more  effectively  in  these  positions.    The  GAO  report  took 
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issue  with  the  long  education  periods,  some  extending  to  3 
years.   Also  the  GAO  noted  that  equivalent  positions  in  the 
Services  were  judged  inconsistently  regarding  the  need  for 
graduate  education.   Some  positions  required  advanced  educa- 
tion and  others  did  not.   Positions  were  validated  as  requir- 
ing advanced  education  only  because  the  officer  in  the 
position  had  an  advanced  degree.   Official  job  descriptions 
requiring  advanced  degrees  were  submitted  for  validation 
which  did  not  address  the  requirement  for  graduate  education 
in  the  job  descriptions.   The  GAO  found  from  its  sample  audit 
of  military  installations  that  officers  with  graduate  educat- 
ion were  not  being  assigned  to  the  positions  which  required 
graduate  education. 

The  GAO  recommended  that  the  Services  use  short  courses 
and  work  experience  as  alternatives  to  full-time  graduate 
education.   Also  they  should  fill  the  positions  with  civil- 
ians where  possible.   DOD  was  requested  to  issue  a  new  policy 
restricting  the  number  of  positions  requiring  a  graduate 
education.   Congress  was  requested  to  control  the  number  of 
military  officers  in  full-time,  fully- funded  graduate  study 
by  limiting  the  funds  available. 

DOD  had  encouraged  the  graduate  education  of  military 
officers.   In  our  society  there  are  generally  accepted  values 
and  benefits  of  graduate  education.   DOD  felt  that  the 
graduate  education  enhanced  the  effectiveness  and  capability 
of  the  officers  and  was  an  important  factor  in  retaining 
them, 
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The  GAO  considered  the  effectiveness  of  long-term, 
full-time  graduate  education  for  DOD  civilians  in  non- 
government institutions  [Ref .  28] .   Long  term  training  was 
considered  training  greater  than  120  days.   The  training 
was  through  courses  at  the  graduate  level  primarily  in 
management,  scientific,  engineering,  or  technical  fields. 

The  DOD  installations  were  critized  because  they  did 
not  follow  existing  regulations.   The  field  activities  had 
not  determined  what  their  near  and  long  term  (5  year)  train- 
ing needs  were  nor  had  they  prioritized  them.   The  GAO 
found  little  evidence  that  there  was  a  plan  to  use  the  new 
skills  of  the  employee  upon  return  from  long-term  training. 
In  addition,  the  field  activities  had:   a)  not  given  much 
thought  to  selecting  employees  for  training  in  areas  where 
there  was  the  greatest  need;  (b)  overemphasized  the  require- 
ment to  obtain  an  academic  degree;  (c)  not  used  any  criteria 
for  selecting  non-government  schools;  and  (d)  not  provided 
for  any  evaluation  of  the  training  to  determine  if  its 
objectives  were  met. 

The  objectives  of  DOD's  support  of  long-term,  full-time 
training  and  education  for  civilian  employees  are  as  follows 
(a)  to  maintain  a  position  of  leadership  in  defense-oriented 
science  and  technology;  (b)  to  provide  opportunities  for 
career  employees  of  promise  to  grow  and  realize  their  full 
potential;  and  Cc)  to  better  enable  employees  to  success- 
fully cope  with  the  complex  problems  of  managing  all  aspects 
of  national  defense  activities. 
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The  Civil  Service  Commission,  DOD,  and  the  Service's 
regulations  emphasize  that  training  qualify  personnel  for 
performance  of  official  duties.   The  accomplishment  of  an 
advanced  degree  should  be  an  incidental  by-product  of  the 
training.  The  GAO  found,  however,  the  field  activities' 
applications  for  long-term  training  implied  that  the  indivi- 
dual was  expected  to  achieve  an  advanced  degree.   Training 
programs  were  frequently  structured  so  that  they  lead  to  an 
advanced  degree,  hence  improving  the  prestige  and  image  of 
the  activity.    Additionally,  GAO  found  that  the  individuals 
applying  for  long-term  training  did  so  out  of  self-interest 
and  their  self-development  did  not  support  the  training 
objective  of  the  activity.   The  individuals  selected  their 
own  course  of  study  and  selected  the  training  institution. 

The  National  Academy  of  Public  Administration  conducted  a 
study  on  military  officer  graduate  education  in  1975  [Ref . 
29] .   This  was  just  prior  to  congressional  action  on  an  0MB 
recommended  70  per  cent  reduction  in  funding  for  full-time, 
fully-funded  graduate  education.   The  panel  concurred  that 
the  military  should  have  educational  background  comparable 
with  their  counterparts  in  the  private  sector  and  government. 
The  existing  system  of  counting  positions  requiring  gradu- 
ate education,  however,  should  be  replaced  with  one  that  sets 
educational  standards  and  integrates  the  need  with  technical, 
organizational,  and  strategic  plans.   It  recommended  the 
Services  utilize  other  programs  such  as  off-duty,  recruitment 
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of  officers  with  advanced  degrees  —  especially  in  fields  with 
shortages  —  and  cooperative  study  (flexibility  for  officers 
to  take  courses  during  normal  working  hours) .   The  Navy  and 
Air  Force  postgraduate  schools  were  encouraged  to  increase 
their  number  of  off-campus  personal  study  courses  so  as  to 
reduce  the  time  in  residency  to  complete  graduate  programs . 
In  order  to  accomplish  more  off-campus  education  with  high 
standards  and  quality,  representatives  of  the  Services  and 
higher  education  should  establish  ground  rules  for  instructor 
selection  and  credit  transfer.   The  report  emphasized  the 
need  for  classroom  situations  with  both  civilian  and  military 
participation  in  the  learning  process. 

The  report  noted  that  the  need  for  graduate  education  is 
obvious  in  the  physical,  mathmatical,  and  engineering  sciences. 
These   disciplines  are  used  in  the  development  and  operation 
of  high  technology  weapons,  transportation,  and  communication 
systems.   In  addition,  advanced  education  is  necessary  in 
biological,  social,  operational,  and  managerial  services  in 
order  to  manage,  deploy,  and  operate  these  systems.   Although 
the  Services  depend  on  civilian  industry  to  provide  the 
talent  to  produce  the  weapons  they  need,  an  officer  corp  less 
educated  than  its  counterparts  will  be  severly  handicapped  in 
its  relations  with  civilians. 

In  a  situation  of  reduced  funding,  technical  advanced 
education  is  the  beneficiary  of  a  majority  of  the  funds  for 
full-time,  fully-funded  education.   It  is  nearly  impossible 
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for  an  officer  to  acquire  an  advanced  degree  in  engineering 
or  science  through  an  off-duty  or  a  partially-funded  program; 
and  it  is  easier  to  couple  specific  technical  and  science 
degrees  with  validated  billets. 

C.   AFTER  WORKING  HOURS 

The  proliferation  of  after  hours  graduate  programs  has 
raised  questions  about  the  quality  of  these  programs.   Most 
installations  have  taken  the  view  that  having  some  graduate 
program  offerings  is  better  than  nothing.   These  programs 
stretch  out  for  several  years  and  have  high  student  drop- 
out rates.   A  majority  of  the  programs  are  management  vice 
technical  oriented  because  most  engineering  schools  have 
been  reluctant  to  offer  complete  off-campus  graduate 
programs . 

There  are  some  limitations  to  graduate  education  conducted 
under  these  conditions.   Most  employees/students  are  at  a  low 
point  in  their  learning  capacity  and  retention  capability 
after  a  day's  work.   The  employee's  class  attendance  is 
frequently  interrupted  by  work-related  travel  or  problems. 
The  classes  consist  primarily  of  work  associates;  therefore, 
there  is  not  the  opportunity  to  experience  the  multiple 
viewpoints  occurring  in  the  campus  environment.   Specific 
programs  are  difficult  to  sustain  due  to  employee  turnover, 
lack  of  recognition  of  credit  for  the  courses  by  other  higher 
education  institutions,  changes  in  the  individual's  education 
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goals,  and  declining  interest  in  the  specific  programs 
offered. 

The  issue  of  "cheap  degrees"  offered  at  military  bases 
was  addressed  by  Gene  Sherron,  an  American  Council  on 
Education  Fellow  [Ref.  30].   He  indicates  that  most  of  the 
programs  are  so  poor  that  these  programs  would  be  classed 
as  diploma  mills  if  given  close  scrutiny  by  the  educational 
community.   At  least  19  colleges  and  universities  from  as 
far  away  as  California  offer  graduate  degree  programs  in 
Washington,  D.C.   Some  major  institutions  refuse  to  partici- 
pate because  it  is  their  position  that  quality  graduate 
education  belongs  on  campus  where  students  have  access  to 
full  time  professors  and  research  facilities. 

So  far  the  accrediting  bodies  and  associations  of  higher 
education  have  not  forced  a  policing  or  correction  in  the 
proliferation  of  advanced  degrees  around  the  nation.   The 
declining  enrollments  and  reduced  college  budgets  have 
accelerated  many  institutions'  quest  for  the  available 
education  dollars  in  the  after-work-hours  market.   The  DOD 
is  partially  to  blame  for  allowing  these  institutions  to 
offer  their  educational  wares  at  military  installations 
without  any  control  of  quality  or  admissions  criteria.   The 
local  government  administrator  normally  does  not  have  an 
advanced  degree  and  is  under  intense  pressure  to  keep 
enrollments  up  and  expenses  down.   Most  programs  use 
part-time  instructors  recruited  locally. 
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D.   THE  CURRICULUM  PROPOSALS 

1-   Naval  Postgraduate  School  Informal  Proposals 

In  the  past  seven  years  the  NPS  faculty  have  held 
informal  discussions  with  T&E  field  activity  representatives 
regarding  the  design  of  a  T&E  curriculum.   Although  it  would 
be  desireable  to  have  a  six  quarter  program,  field  activity 
management  is  strongly  opposed  to  providing  full-time 
training  for  more  than  one  year.   The  military  probably 
would  not  have  difficulty  with  this  point,  since  most  of  the 
sponsored  NPS  curricula  are  six  quarters  and  longer. 
Table  I  summarizes  informal  NPS  past  proposals  for  a  T&E 
curriculum.   These  proposals  have  a  high  probability  and  statis- 
tics content  and  a  strong  operations  research  orientation. 
The  thesis  requirement  creates  a  heavy  workload  for  the 
four  quarter  curriculum.   Since  the  students  most  probably 
will  have  written  technical  reports  prior  to  attending  the 
NPS  and  will  write  technical  reports  upon  return  to  their 
work  place,  the  thesis  requirement  could  be  waived  without 
reducing  the  quality  of  the  graduate  education.   The  NPS 
retains  the  control  of  the  quality  of  the  graduate  education 
via  having  the  student  in  full  time  residence  at  NPS;  and 
the  test  activity  would  reinforce  the  quality  of  the  educa- 
tional experience  through  ensuring  that  the  graduate  has  the 
opportunity  to  write  a  formal  technical  report  upon 
completion  of  the  NPS  academics. 
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The  NPS  "strawman"  curricula  are  informal  proposals  and 
do  not  really  address  responsibility  for  the  program.   In 
order  to  ensure  that  there  is  a  linking  of  engineering, 
management,  and  operations  research  disciplines,  some 
thought  should  be  given  by  the  NPS  administration  to  consider- 
ing this  curriculum  as  a  Masters  of  Science  in  Engineering- 
Management  degree  program.   Although  there  appears  to  be 
adequate  field  activity  interest  in  a  T&E  oriented  curriculum 
at  NPS,  it  has  not  happended  for  numerous  reasons.   Among 
these  is  a  lack  of  high  level  Department  of  the  Navy  interest 
in  a  T&E  program  and  the  availability  of  the  resources 
required  by  NPS  to  implement  the  program. 

2 .   Pacific  Missile  Test  Center 

The  Pacific  Missile  Test  Center  (PMTC)  Point  Mugu, 
California  has  advocated  the  establishment  of  an  advanced 
degree  in  T&E  since  the  mid  1970 's.   In  1974  it  was  able  to 
establish  an  innovative,  off-campus  graduate  education  pro- 
gram in  engineering  with  the  cooperation  of  the  California 
State  University,  Northridge  [Ref.  31,32].   This  program. 
Technical  Professional  Program,  was  developed  around  the  new 
engineering  graduate  employeed  by  PMTC. 

The  objectives  were  to  prepare  the  engineer  for  roles 
as  a  technical  specialist  in  T&E  and  to  develop  the  engineer's 
skill  and  competence  to  plan  and  analyze  large  scale  tests. 
The  students  attended  classes  half-time  and  worked  half-time. 
They  were  expected  to  complete  the  program  in  one  calendar 
year.   The  work  projects  were  designed  to  support  the 
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classroom  studies.   Certainly,  this  was  a  bold  attempt  to 
bring  graduate  education  into  the  real  world  to  support  the 
needs  of  the  practioner.   Some  of  the  weak  points  of  this 
program  appear  to  be  the  employee  counselor,  and  a  work 
dossier  which  was  maintained  by  the  student  and  graded  as 
satisfying  course  requirements.   This  program  placed  much 
responsibility  on  the  PMTC  employees  to  act  as  pseudo- 
educators  and  to  ensure  that  the  students  had  project  exper- 
ience equivalent  to  an  on-campus  graduate  level  learning 
situation.   The  Technical  Professional  Program  was  conducted 
for  one  year  ending  in  1976. 

The  PMTC  has  drawn  from  the  experience  gained  from  the 
Technical  Professional  Program  and  has  proposed  the  Test  and 
Evaluation  Intern  Program  [Ref ,  33] .   This  is  a  two  year 
program  designed  to  develop  a  T&E  Professional  Engineer. 
Academic  instruction  is  provided  by  California  State 
University,  Northridge  and  University  of  California,  Santa 
Barbara.   The  program  participants  would  spend  the  six 
semesters  of  the  program  at  various  test  sites  performing 
assignments  in  T&E  of  components,  subsystems,  and  systems. 
Formal  course  work  would  be  attended  during  part  of  the 
work  day.   The  courses  are  listed  in  Table  II.   This 
program  requires  substantial  support  from  the  Navy  T&E 
community  to  progress.   It  is  a  commendable  effort  because 
it,  like  the  predecessor  program.  Technical  Professional 


48 


vO 


u 

o 

a 

a» 

(U 

cd 

> 

•H 

iH 

■U 

cd 

O 

c 

(U 

u 


* 

c 

X5 

0) 

^ 

<u 

/— \ 

in 

o 
u 

H 

1 

CO 
u 
C 
(U 
•H 

> 

•H 
4.) 

u 

(U 

rH 

Ed 

Cd 

1 

00 
•H 
CO 
CO 

< 

e 
cu 

4J 
CO 

>^ 

c 

(U 

c 
o 

CI. 

P5 

<T 

CO 

»-i 

e 

o 

c 

o 

o 

0) 

(U 

C 

o 

4-t 

u 

o 

}-l 

04 

< 

•H 

•H 

o 

H 
H 

w 

1 

1 

CJ 

<u 

CO 

Ix] 

&H 

a 

CO 

M 

•^ 

00 

5 

J 

2 

•H 

M 

CO 

C 

/-^^ 

4J 

ffl 
< 

M 

00 

<u 

(U 

cs 

•H 

CO 

UU 

c 

a 

CO 

o 

r-l 

4-1 

B 

o 

bj 

o 

OJ 

•H 

T3 

£3 

4) 

C/3 

Eh 

w 

OS 

4-1 

c: 

^ 

cu 

4J 

t« 

CO 

M 

va 

CO 

CO 

U 

g 

CO 

M 

Eh 

s 

CO 

e 

n: 

O 

2 

>^ 

o 

<u 

00 

c 

!>i 

3 

00 

CO 

U 

■u 

c 

o 

O 

CO 

•H 

Xi 

*v 

CO 

•H 

•H 

<4-l 

4-1 

w 

CO 

3 

cu 

Eh 

>. 

>^ 

4-) 

fl 

CO 

ca 

CO 

en 

•H 

S 

U3 

cu 

CO 

M 

Q 

H 

< 

S»' 

X 

04 

CO 

4J 

u 

CO 
4J 

V3 

M 
O 

CO 
(U 

•H 

14-1 

c 

a 

c 

iH 

^4 

o 

0) 

^ 

cu 

cu 

•H 

CN 

(U 

CO 

e 

V4 

s 

c 

CO 

0) 

a 

G 

•H 

o 

c 

0 

CO 

c 

OJ 

00 

U 

:s 

00 

CL 

•H 

•H 

u 

•H 

0) 

•H 

e 

S 

00 

CO 

CO 

Ou 

w 

CO 

o 

c 

>% 

0) 

X 

ca 

CO 

CJ 

M 

M 

Q 
o 

w 

H 

<: 

>-• 

o 

4J 

CO 
M 

CO 

>4> 

4-) 

Ed 

O 

(U 

H 

c 

<^ 

U 

.H 

0) 

H 

•H 

iH 

O. 
O 

c 

•H 

>> 
> 

CO 

o 

00 
CO 

c 

CO 

> 

CO 

< 

1 

•K 

M 

•• 

(U 

Cd 

4J 

c-i 

(0 

o 

0) 

z 

Q) 

CO 

49 

Program,  requires  close  working  cooperation  between  the 
educational  institution,  the  student,  and  the  participating 
T&E  activity  to  enrich  the  learning  experience  of  graduate 
education. 

E.   CHARACTERISTICS  OF  PROFESSIONS 

DOD  high  level  officials  have  been  reluctant  to  acknow- 
ledge the  existence  of  T&E  as  being  a  distinct  technical 
discipline.   This  is  understandable  because  most  DOD  personnel 
are  not  familiar  with  the  T&E  process  and  its  role  in  the 
acquisition  of  weapon  systems.   The  President's  Blue  Ribbon 
Panel  Report  on  OT&E  supports  this  observation  [Ref .  34] . 
Government  personnel,  primarily  in  the  development  test 
activities,  have  been  the  primary  spokesmen  for  calling 
attention  to  the  evolution  of  a  T&E  engineering  discipline. 
Although  they  have  asked  academia  to  develop  formal  courses 
and  curriculum  to  support  their  needs,  engineering  institutions 
have  paid  little  attention  to  the  requests  of  the  T&E  community 
A  review  of  the  characteristics  of  professions  is  in  order  to 
help  explain  the  lack  of  recognition  of  T&E  as  an  engineering 
discipline. 

The  difference  between  occupations  and  professions  can  be 
viewed  as  a  continuum.   The  occupations,  such  as  carpentry  and 
brick- laying,  are  on  the  lower  end  and  medicine  and  law  on  the 
upper  end  of  the  continuum.   A  number  of  attributes  have  been 
defined  which  are  normally  associated  with  a  profession 
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[Ref.  35].   These  are:   (1)  a  systematic  body  of  theory;  (2) 
professional  authority;  (3)  sanctions  by  the  community;  (4) 
a  regulative  code  of  ethics;  and  (5)  a  professional  culture. 
The  U.  S.  Census  Bureau  in  its  occupational  classification 
includes  engineers  in  the  professional  category.   Again,  look- 
ing at  the  continuum  of  professions,  the  engineering  profes- 
sion would  be  considered  below  the  law  and  medical  professions. 
The  environment  in  which  most  engineers  work  differs  from 
other  professionals  [Ref.  25].   The  T&E  engineers,  for  example, 
differ  as  follows:   (1)  they  do  not  choose  their  clients;  (2) 
they  do  not  have  on  the  job  autonomy;  (3)  they  do  not  have  to 
obtain  a  recognized  postgraduate  educational  certification; 
and  (5)  they  do  not  have  a  strong,  professionally-backed  set 
of  standards  and  ethics. 

The  profession  which  has  really  arrived  is  noted  by  certain 
characteristics  which  enables  it  to  exercise  social  control 
over  its  community.   What  evolves  then  is  the  professional 
community  within  the  larger  social  community  [Ref.  35].   The 
professional  community  members  tend  to  take  on  the  following 
characteristics:   (1)  bound  by  sense  of  identity;  (2)  once  in 
the  profession,  continue  for  their  whole  career;  (3)  share 
values  in  common;  (4)  role  definitions  which  are  agreed  to  by 
both  members  and  non-members;  and  (5)  take  on  a  common  language 
which  is  only  partially  understood  by  non-members.   In  addition 
the  community  has  power  over  its  members;  its  imposed  social 
limits  are  reasonably  clear;  and  it  controls  the  composition 
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of  the  community  in  the  future  through  selection  of  profes- 
sional trainees  and  the  training  processes.   The  T&E  communi- 
ty has  not  demonstrated  that  it  has  moved  to  the  point  where 
it  exhibits  any  of  these  characteristics. 

The  knowledge  explosion  has  created  specialization  within 
the  professions.   Technology  appears  to  be  increasing  almost 
exponentially  creating  a  broad  technical  base  and  many  combina- 
tions of  knowledge.   Professional  specialization  is  partly 
responsible  for  creating  the  technology  expansion  which  in 
turn  has  increased  the  amount  of  specialization.   The 
specialist,  for  example  the  electronics  engineer,  may  become 
obsolete  because  his  specialty  may  still  change  too  fast  for 
an  individual  to  keep  current.   His  specialty  might  be  super- 
seded by  advances  in  knowledge  that  make  his  knowledge  or 
skill  unnecessary  or  inadequate.   To  an  increasing  degree  a 
professional  career  must  be  regarded  as  a  continuous  learning 
experience.   The  call  by  the  DOD  T&E  community,  for  recogni- 
tion as  a  unique  engineering  specialty,  is  an  expression  of 
this  phenomenon. 

F.   FOCAL  POINT  FOR  T&E  EDUCATION? 

The  DOD  sponsors  the  Defense  Systems  Management  College 
(DSMC) ,  Fort  Belvoir,  Virginia  to  train  prospective  program 
managers.   The  DSMC  primarily  provides  training  to  military 
and  civilian  employees  of  the  DOD  who  will  be  employed  in  the 
acquisition  of  weapon  systems.  There  is  also  limited  participa- 
tion by  contractors.   The  DSMC,  therefore,  has  become  a  focal 
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point  for  the  discussion,  disemmination,  and  training  in 
program  management  principles.   Problems  with  the  DOD  acquisi- 
tion cycle  are  aired  here  and  possible  solutions  are  discussed 
in  a  classroom  environment.   Students  are  instructed  in 
problems  and  future  trends  in  program  management. 

High  level  DOD  management  officials  participate  in  DSMC 
sponsored  programs  and  publications.   Thus,  through  this 
focal  point  improvements  in  DOD  management  techniques  can  be 
proposed,  assessed,  and  discussed  openly.   The  resulting 
benefits  to  DOD,  by  providing  program  management  training  to 
military,  civilian  and  contractor  personnel  are:   (1)  provides 
a  common  ground  to  communicate  the  DOD  policy  on  acquisition 
management  techniques;  (2)  prepares  high  level  military  and 
civilian  personnel  who  will  be  assigned  to  program  offices 
for  the  rigors  of  program  management;  (3)  introduces  these 
personnel  to  the  political  aspects  of  the  acquisition  of 
weapon  systems;  and  (4)  sharing  of  lessons  learned  from  case 
studies  of  previous  acquisition  programs. 

The  T&E  community  has  no  counterpart  to  the  DSMC.   T&E 
has  not  been  institutionalized  in  DOD.   There  is  no  recognized 
focal  point  in  the  T&E  community  where  the  Services  can 
exchange  lessons  learned,  share  methodologies  and  techniques, 
build  a  knowledge  base,  disseminate  information,  conduct 
research,  and  encourage  cooperation.   With  minimal  interference 
from  DOD  each  of  the  Services  have  developed  separate  approach- 
es to  T&E.   The  Joint  Logistics  Commanders  work  to  improve 
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the  participation  of  each  of  the  Services  in  joint  service 
acquisition  programs.   Ad  hoc  working  groups  are  formed  to 
study  specific  problems  and  to  improve  the  conduct  of  joint 
Service  programs.   The  Range  Commanders  Council,  chaired  by 
the  Director  of  Defense  T&E,  occasionally  sponsors  ad  hoc 
working  groups  to  address  T&E  issues.   Only  the  T&E  activities 
designated  as  part  of  the  Major  Range  and  Test  Facility  Base 
(MRTFB)  participate  in  this  forum.   They  are  listed  in 
Appendix  A,  Table  I. 

The  DOD  through  the  DDT&E  should  sponsor  a  study  or  task 
the  Services  to  compile  the  available  knowledge  on  T&E.   It 
should  inventory  the  military  and  civilian  skills  required  to 
conduct  T&E. 

G.   TAILORING  T&E  EDUCATION 

There  is  increasing  evidence  that  greater  cooperation 
between  academic  institutions  and  the  worksite  for  graduate 
education  can  enrich  the  learning  experience  for  all  the 
participants  [Refs.  33,37].   Academia  receives  immediate  feed- 
back for  an  evaluation  of  the  curriculum  and  the  needs  of  the 
student  relative  to  the  marketplace.   The  T&E  education  needs 
are  a  variable  as  shown  in  Figure  1.   The  more  senior  the 
DOD  official  involved  in  T&E  the  less  are  his  T&E  education 
requirements;  and,  therefore,  T&E  seminars  of  a  few  days  would 
be  appropriate. 

The  new  engineer  graduates  have  been  high  in  analytical 
skills  and  low  in  synthesizing  and  design  skills.   The  early 
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T&E  experience  is  gained  through  assisting  more  experienced 
engineers.   Through  a  combination  of  employer  needs  and  the 
engineer's  interests  a  degree  of  specialization  evolves.   The 
engineer,  for  example,  after  involvement  in  several  test 
projects  with  a  given  type  aircraft  to  determine  its  per- 
formance characteristics,  may  become  aligned  with  one 
specialty  of  aerodynamics.   He  will  then  pursue  technical 
updating  in  expectation  of  receiving  a  graduate  degree.   Most 
data  shows  engineers  to  be  stable  and  90%  are  expected  to 
remain  in  their  field  for  five  years  after  graduation  [Ref. 
25] .   Provided  the  engineer  stays  at  a  DOD  T&E  activity,  he 
will  observe  that  more  of  his  time  is  occupied  with  manage- 
ment related  matters.   This  situation  also  begins  to  surface 
the  deficiencies  in  the  engineering  education  for  dealing 
with  the  social  and  human  problems  which  are  more  difficult 
to  bound  than  analytical  problems.   At  this  point  a  decision 
is  required  on  pursuing  technical  management.   Leaving  the 
T&E  activity,  pursuing  a  technical  specialist  career  or 
developing  managerial  skills  are  the  main  options  available. 
Some  T&E  activities  have  been  more  successful  than  others  in 
developing  a  dual  career  advancement  program;  however,  this 
still  remains  a  critical  decision  point  for  most  engineers. 
The  evolution  of  education  needs  as  shown  in  Figure  1  gives 
evidence  of  the  need  for  flexibility  in  the  concept  of  T&E 
education.   The  needs  of  the  specific  T&E  activity  should  be 
matched  with  the  needs  of  the  individual  to  perform  his  duties. 
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It  is  at  this  point  T&E  education  can  have  a  recognizeable 
form.   The  types  of  education  which  are  shown  in  Figure  2 
depict  in  general  the  needs  of  T&E  personnel.   The  point  at 
which  these  education   needs  change  are  peculiar  to  each 
individual  and  T&E  activity.   It  does  provide,  however,  an 
indication  of  the  range  of  education  required.   The  technical 
report  writing  and  test  planning  need  to  be  reinforced  and 
integrated  into  the  specific  technical  education  courses. 
The  repetition  and  reinforcement  of  these  basic  skills  are 
critical  in  communicating  T&E  results  concisely. 

In  the  case  of  the  military  officer  entering  T&E  of  air- 
craft weapons  systems  a  different  education  evolution  occurs. 
The  operating  skills  of  the  officer  are  used  to  support  the 
T&E  process.   For  example,  it  depends  on  which  T&E  activity 
he  is  assigned  to  as  to  whether  or  not  he  will  receive 
specific  training  at  the  Naval  Test  Pilot  School.   This 
training  develops  the  skills  to  conduct  engineering  flight 
testing.   The  military  officer  T&E  education  needs  are 
normally  different  than  those  for  the  engineer.   He  may  only 
spend  two  years  in  T&E  and  is  more  concerned  with  operating 
skills  and  management  education.   He  may  have  already  attended 
the  Naval  Postgraduate  School  and  received  graduate  education 
in  engineering  prior  to  coming  to  the  T&E  activity.   The 
officer  who  has  a  designated  subspecialty  as  an  aviation 
engineering  duty  officer  normally  has  an  advanced  technical 
degree  prior  to  coming  to  the  T&E  activity.   His  education 
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needs  would  lie  more  to  the  right  in  Figures  1  and  2.   If  a 
T&E  curriculum  were  available,  for  example  at  the  NPS,  then 
the  officers  completing  the  curriculum  would  be  more  effec- 
tive at  their  jobs  sooner  and  require  less  on-the-job  train- 
ing time.   The  officers  interviewed  in  reference  [2]  indicated 
that  knowledge  about  T&E  principles  and  program  management 
would  have  helped  their  effectiveness  in  their  jobs  and 
reduced  their  on-the-job  training. 

The  need  for  T&E  education  so  far  has  come  from  the 
personnel  assigned  to  the  T&E  activities.   The  T&E  body  of 
knowledge  is  disbursed  throughout  the  DOD  T&E  activities. 
The  Director  of  Defense  T&E  (DDT&E)  support  is  needed  for  the 
support  of  the  activities'  efforts  to  define  their  graduate 
education  requirements.   DDT&E  could  contract  for  a  study 
with  industry,  a  nongovernment  academic  institution,  or  a 
military  postgraduate  school  on  the  issues  involving  T&E 
education.   Some  of  the  parameters  this  study  could  consider 
are:   (1)  the  bounds  and  content  of  the  T&E  body  of  knowledge; 
(2)  a  focal  point  for  T&E  knowledge;  (3)  T&E  education;  (4) 
military  careers  in  T&E;  (5)  future  trends  in  T&E;  and  (6) 
rotation  of  civilian  and  military  personnel  between  T&E 
activities . 

The  Congress  has  not  always  been  pleased  with  the  Services' 
management  of  T&E.   Congress  would  be  supportive  of  DDT&E 
efforts  to  exercise  management  improvement  of  the  T&E  process. 
Congress  has  expressed  concerns  in  the  past  that  the  Service 
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filter  T&E  results  of  major  weapon  systems  in  order  to  enhance 
the  chances  Congress  will  provide  funding  for  procurement. 
Various  studies  have  critisized  the  delays,  duplications,  and 
cost  of  T&E.   These  are  symptoms  which  have  been  treated  by 
numerous  directives  and  instructions  and  even  establishment 
of  the  DDT&E.   Possibly,  through  T&E  education,  developing  a 
body  of  T&E  knowledge,  and  establishing  a  DOD  focal  point  for 
distribution  of  this  knowledge  the  significance  and  role  of 
T&E  can  be  fairly  assessed. 
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IV.   CONCLUSIONS  AND  RECOMMENDATIONS 

A.   CONCLUSIONS 

1.  General 

The  hypothesis  is  not  presently  supportable.   There 
is  little  evidence  to  justify  the  establishment  of  a  graduate 
TStE  curriculum.   There  is  a  real  need,  however,  for  a  compila- 
tion of  the  information  available  on  T&E  in  the  DOD.   It  is 
the  DDT&E  responsibility  to  define,  support,  and  initiate 
this  effort. 

2.  Specific 

The  DOD  officials,  in  attempting  to  solve  the  problems 
of  the  acquisition  of  major  weapon  systems,  identified  weak- 
nesses in  the  role  of  T&E.   The  solutions  have  been  to  change 
organizational  structure  and  to  change  the  timing  and  participa- 
tion of  T&E  personnel  in  the  acquisition  cycle.   The  problems 
in  T&E,  however,  seem  more  fundamental.   The  area  of  T&E 
education  has  not  been  considered  by  high  level  DOD  officials 
as  relevant  to  improving  the  effectiveness  of  T&E, 

The  Services  have  developed  different  implementations 
of  the  concept  of  independent  testing  as  required  by  DOD 
directives.   These  differences  raise  policy,  procedural  and 
philosophical  issues,  when  the  Services  are  required  to  par- 
ticipate in  joint-service  acquisition  programs.   The  same  sort 
of  situation  also  exists  in  combined  intra-Service  testing. 


61 


The  trend  for  the  future,  in  order  to  decrease  duplication 
of  efforts  and  acquisition  costs  and  schedules,  is  to  increase 
the  number  of  joint-Service  programs  and  of  intra-Service 
combined  tests.   This  points  to  a  need  for  increased  high 
level  DOD  officials'  attention  to  the  education  and  training 
needs  of  T&E  personnel  to  reduce  communication  problems. 

It  is  not  generally  accepted  or  recognized  by  high 
level  DOD  officials  that  TSeE  requires  unique  education.   T&E 
in  support  of  the  acquisition  cycle  is,  however,  unique  to 
DOD.   The  body  of  knowledge,  professional  society  representa- 
tion, and  community  characteristics  do  not  exist  in  a  highly 
visible  form.   The  DOD  T&E  community  is  not  well  organized 
or  well  represented.   There  is  no  consensus  on  what  a  T&E 
specialist  is.   Most  engineers  and  officers  would  not,  if 
asked,  claim  to  be  a  T&E  specialist  but  would  instead  identify 
with  the  engineering  discipline  in  which  they  earned  their 
degree,  for  example  electrical  engineering. 

Academic  institutions  have  not  generally  embraced  the 
concept  of  T&E  as  a  specialty.   There  has  been  little  academic 
interest  in  establishing  T&E  courses  or  developing  faculty  to 
support  a  T&E  curriculum.   The  educational  parameters  necessary 
to  support  T&E  seem  to  point  more  toward  a  blend  of  engineering 
and  management  education. 

The  T&E  personnel  are  not  very  mobile.   There  has  been 
very  little  effort  in  DOD  to  utilize  T&E  personnel  across 
Service  lines  or  intra-Service  between  test  agencies.   The 
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opportunities,  therefore,  for  cross-pollenization  of  T&E 
skills,  policies,  test  methods  and  lessons  learned  have  been 
minimal.   Government  personnel  policies  do  not  provide  for 
the  transfer  of  T&E  personnel  to  programs  where  their  talents 
are  most  needed.   The  evolution  and  identity  of  the  T&E 
Specialists  with  special  education  requirements  can  only  be 
impeded  by  this  policy. 

The  Naval  Aviation  Executive  Institute  (NAEI)  program 
in  cooperation  with  the  Naval  Postgraduate  School  could  be 
the  leader  in  the  DOD  for  defining  and  supporting  the  graduate 
education  needs  of  T&E  personnel.   Increased  cooperation  and 
flexibility  are  required  between  the  NAEI,  NPS,  and  the  Navy's 
T&E  agencies.   The  uncertainties,  of  future  civilian  T&E 
personnel  participation  in  NPS  provided  education,  could  be 
resolved  through  improved  communications  of  the  needs  of  each 
of  the  participating  organizations. 

B .   RECOMMENDAT IONS 

1.   The  Director  Defense  T&E  (DDT&E) 
The  DDT&E  should: 

(1)  establish  a  focal  point  for  the  compilation  and 
dissemination  of  T&E  information. 

(2)  initiate  a  study  to  define  the  DOD  T&E  body  of 
knowledge. 

C3)  sponsor  an  annual  seminar  on  DOD  T&E. 
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(4)  develop  a  seminar  course  consisting  of  20-40  hours 
of  instruction  on  DOD  T&E  for  high  level  DOD 
officials . 

(5)  assess  the  T&E  related  education  needs  of  the 
T&E  agencies  and  prepare  a  plan  to  support  these 
needs. 

2.  The  Services 

Each  of  the  Services  should; 

(1)  compile  its  existing  body  of  T&E  knowledge. 

(2)  review  the  educational  needs  of  its  T&E  personnel. 

(3)  define  career  paths  for  T&E  personnel  which  includes 
involvement  in  acquisition  management  decision 
making. 

3.  NFS,  NAEI,  and  the  Navy's  RDT&E  Agencies 
Representatives  from  these  activities  should  meet  to 

review  the  trends  in  graduate  education,  and  the  need  to 
involve  the  NFS  more  closely  in  the  integration  of  education 
with  the  education  needs  of  the  T&E  practioners. 
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APPENDIX  A 


DOD  5000.1 
Major  System 
Acquisition 


0MB  CIRCULAR 
A-109 
Major  System  Acquisitions 


DOD  5000.2 

Major  System 

Acquisition  Procedures 


i 


OPNAVINST  396  0.10A 
Test  and  Evaluation 


NAVMATINST  3960.  6A 
Test  and  Evaluation 


NAVAIRINST  3960.2A 
Test  and  Evaluation 


TEMP 


DOD  5000.3 
Test  St   Evaluation 


(Test  and  Evaluation  Master  Plan) 


I 


Figure  2.   The  Flow  of  Instructions  to  Develop  a  TEMP 
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APPENDIX  A 


TABLE  I 


DEPARTMENT  OF  DEFENSE  MAJOR  RANGE 
AND  TEST  FACILITY  BASES 


ARMY 


Cold  Regions  Test  Center 
Fort  Greely,  Alaska 

Tropic  Test  Center 

Fort  Clayton,  Canal  Zone 

Yuma  Proving  Ground 
Yuma,  Arizona 

Jefferson  Proving  Ground 
Madison,  Indiana 


White  Sands  Missile  Range 
New  Mexico 

Kwajalein  Missile  Range 
Marshall  Islands,  Pacific 

Electronic  Proving  Ground 
Fort  Huachuca,  Arizona 

Dugway  Proving  Ground 
Salt  Lake  City,  Utah 

Aberdeen  Proving  Ground 
Aberdeen,  Maryland 


NAVY 


Pacific  Missile  Test  Center  Naval  Air  Engineering  Center 


Point  Mugu,  California 

Atlantic  Undersea  T&E 

Center 
Andros  Island,  Bahamas 

Naval  Air  Test  Center 
Patuxent  River,  Maryland 

Naval  Air  Propulsion  Test 

Center 
Trenton,  New  Jersey 


Lakehurst,  New  Jersey 


Naval  Weapons  Center 
China  Lake,  California 

Atlantic  Fleet  Weapons  Training 

Facility 
Roosevelt  Roads,  Puerto  Rico 


AIR  FORCE 


Space  &  Missile  Test  Center   Flight  Test  Center 
Vandenberg  AFB,  California   Edwards  AFB ,  California 


Eastern  Test  Range 
Patrick  AFB,  Florida 


Armament  Development  &  Test 

Center 
Eglin  AFB,  Florida 
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AIR  FORCE  (Continued) 

Satellite  Control  Facility   Air  Defense  Weapons  Center 
Sunnyvale,  California        Tyndall  AFB,  Florida 

Tactical  Fighter  Weapons     Arnold  Engineering  Development 

Center  Center 

Nellis  AFB,  Nevada  Tullahoma,  Tennessee 

4950th  Test  Wing 
Wright-Patterson  AFB,  Ohio 
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